Algorithmic Differentiation Through Automatic Graph Elimination Ordering (ADTAGEO)

Algorithmic Differentiation Through Automatic Graph Elimination Ordering (ADTAGEO) is based on the principle of Instant Elimination: Instead of storing a representation of the complete Computational Graph (tape-based approach of ADOL-C [1]) during the execution of the code to be differentiated and applying some elimination sequence afterwards, we dynamically maintain a DAG representing only active variables that are alive at any point in time. Whenever an active variable is deallocated or its value is overwritten the corresponding vertex in the Live-DAG will be eliminated immediately by the well-known vertex elimination rules [2]. Consequently, the total memory requirement is similar to that of the sparse forward mode. However, if local variables are destructed in the opposite order of their construction (as in C++), single assignment sequences of a code are in effect differentiated in reverse mode. Especially if compiler generated temporaries are destroyed in this way Instant Elimination leads to statement level reverse mode of ADIFOR [3] naturally. More generally, the user determines the elimination order intentionally (or unintentionally) by the order in which he declares variables. This opens the opportunity to overcome the categorical separation between forward and reverse mode by creating hybrid modes by splitting the computation of derivatives in forward and reverse parts. So a tradeoff can be achieved between the time consuming forward mode and the memory intensive reverse mode of AD. If the subgraph spanned between the independent and dependent variables becomes bipartite by Instant Elimination than the desired derivatives are full accumulated in the Live-DAG. Thus no explicit initiation of sweeps is required. Note that a bipartite Live-DAG can be achieved often easily by restructuring the source code: Define a toplevel routine with independents and dependents only as arguments and all other active variables locally in that top-level routine. Then Instant Elimination makes sure that all vertices located between independent and dependent variables will be removed on termination of the top-level routine execution.